Video endpoints and related methods for transmitting stored text to other video endpoints

ABSTRACT

A video endpoint and related methods are disclosed. A video endpoint includes a control circuit configured to provide a user interface configured to enable a user to communicate with another user of another video endpoint with text during video communications with the other user. The user interface is also configured to enable the user to select from text strings stored in a database. A method of operating the video endpoint includes transmitting data corresponding to a selected one of the text strings stored in the database to the other video endpoint during the video communications responsive to the user selecting the selected one of the text strings. A method of transforming a computing device into a video endpoint includes storing computer-readable instructions directed to performing the functions of the control circuit on one or more servers, and transmitting the computer-readable instructions to the computing device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/543,706, filed Nov. 17, 2014, U.S. Pat. No. 9,473,627 (Oct. 18, 2016)which is a continuation-in-part of U.S. patent application Ser. No.14/075,968, entitled “APPARATUSES AND METHODS FOR OPERATING ACOMMUNICATION SYSTEM IN ONE OF A TONE MODE AND A TEXT MODE,” filed Nov.8, 2013, U.S. Pat. No. 9,185,211 on Nov. 11, 2015, the disclosure ofeach of which is hereby incorporated in its entirety herein by thisreference.

TECHNICAL FIELD

The disclosure relates to conveying information from data signals from acommunication device to a far end communication device according to aplurality of different modes. More specifically, the disclosure relatesto operational modes for conveying information from data signals duringcommunication sessions involving audibly-impaired people.

BACKGROUND

Traditional communication systems, such as standard and cellulartelephone systems, enable verbal communications between people atdifferent locations and dual tone multiple frequency (DTMF)communications for transmitting information to machines via tones.Communication systems for audibly-impaired (i.e., hearing-impaired,speech-impaired, or a combination thereof) individuals may also enablenon-verbal communications instead of, or in addition to, verbalcommunications. Some communication systems for audibly-impaired peopleenable communications between communication devices for audibly-impairedpeople and communication systems for audibly-capable users (e.g.,standard telephones, cellular telephones, etc.). For example, a videorelay service (VRS) may provide speech to sign language translationservices, and sign language to speech translation services for acommunication session between a video phone for an audibly-impaired userand a traditional telephone for an audibly-capable user.

Some communication systems for audibly-impaired people enablecommunications between communication devices for audibly-impaired people(e.g., video endpoint to video endpoint communications). Thesecommunication systems may operate with or without intervention from avideo relay service, depending on the audible impairments/capabilitiesand non-audible capabilities (e.g., ability to understand and/or sign ina sign language) of the parties operating the communication devices foraudibly-impaired people.

Conventionally, the audibly-impaired user communicates with a callassistant (e.g., communicating via sign language), and then the callassistant conveys the messages to the far-end user. The call assistantmay act as a translator for both the audibly-impaired user (e.g.,communicating via sign language) and the far-end user (communicating viaspeech communication). If the far-end user is not a live person (e.g.,an automated phone menu system), the call assistant may interpret theautomated menu options to the audibly-impaired user via sign language.The audibly-impaired user may respond by using sign language to instructthe call assistant which selection to choose. The call assistant thenmakes the appropriate selection. For example, the call assistant mayspeak the message from the audibly-impaired user if the automated phonemenu system is configured to recognize and interpret voice.Alternatively, the far-end user (e.g., automated phone menu system) maybe configured to recognize DTMF signals. Therefore, the call assistantmay use a numeric keypad configured to send such DTMF signals to theautomated phone system. The inventors have appreciated that, at times,the audibly-impaired person may desire to convey information through thecommunication device to the far-end user through methods other than bysign language communication, which functionality is not offered byconventional VRS systems.

BRIEF SUMMARY

In some embodiments, the disclosure comprises a video endpoint includingvideo equipment configured to enable an audibly-impaired user to engagein video communications with another user of another video endpoint. Thevideo endpoint also includes at least one control circuit configured toprovide a user interface to the audibly impaired user. The userinterface is configured to enable the audibly-impaired user tocommunicate with the other user with text during the videocommunications with the other user. The user interface is alsoconfigured to enable the audibly-impaired user to select from textstrings stored in a database. The at least one control circuit is alsoconfigured to transmit a selected one of the text strings to the othervideo endpoint for presentation to the other user.

In some embodiments, the disclosure comprises a method of operating avideo endpoint for audibly-impaired users. The method includes engagingin video communications with another video endpoint, and enabling anaudibly-impaired user of the video endpoint to select from text stringsstored in a database. The method also includes transmitting datacorresponding to a selected one of the text strings stored in thedatabase to the other video endpoint during the video communications fordisplay by the other video endpoint responsive to the audibly-impaireduser selecting the selected one of the text strings.

In some embodiments, the disclosure comprises a method of transforming acomputing device into a video endpoint. The method includes storingcomputer-readable instructions on one or more storage devices of one ormore servers configured to communicate with the computing device. Thecomputer-readable instructions are directed to instructing the computingdevice to provide a user interface to an audibly-impaired user. The userinterface is configured to enable the audibly-impaired user tocommunicate with another user of another video endpoint with text duringvideo communications with the other user, and enable theaudibly-impaired user to select from text strings stored in a database.The computer readable instructions are also directed to instructing thecomputing device to transmit a selected one of the text strings to theother video endpoint for presentation to the other user during the videocommunications. The method also includes transmitting thecomputer-readable instructions to the computing device.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a simplified block diagram of a communication system accordingto an embodiment of the disclosure;

FIG. 2 is a simplified block diagram of the first communication deviceof FIG. 1;

FIG. 3 is a simplified block diagram of a computing device for the callassistant of the relay service of FIG. 1;

FIG. 4 is a flowchart illustrating a method of operating thecommunication system according to an embodiment of the disclosure;

FIG. 5 is a flowchart illustrating a method of operating a communicationsystem according to an embodiment of the disclosure;

FIG. 6 is a simplified block diagram of a communication system accordingto an embodiment of the disclosure;

FIG. 7 is a block diagram illustrating entries in a database of thecommunication system of FIG. 6;

FIGS. 8A through 8D are examples of a simplified graphical userinterface of a video endpoint of the communication system of FIG. 6;

FIG. 9 is a simplified flowchart illustrating a method of operating thevideo endpoint of FIG. 6;

FIG. 10 is a simplified block diagram of a communication system fortransforming a computing device into a video endpoint of FIG. 6; and

FIG. 11 is a simplified flowchart illustrating a method of transformingthe computing device of FIG. 10 into a video endpoint of FIG. 6.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings which form a part hereof, and in which is shown byway of illustration specific embodiments in which the disclosure may bepracticed. These embodiments are described in sufficient detail toenable those of ordinary skill in the art to practice the disclosure. Itshould be understood, however, that the detailed description and thespecific examples, while indicating examples of embodiments of thedisclosure, are given by way of illustration only and not by way oflimitation. From this disclosure, various substitutions, modifications,additions rearrangements, or combinations thereof within the scope ofthe disclosure may be made and will become apparent to those of ordinaryskill in the art.

In accordance with common practice the various features illustrated inthe drawings may not be drawn to scale. The illustrations presentedherein are not meant to be actual views of any particular apparatus(e.g., device, system, etc.) or method, but are merely idealizedrepresentations that are employed to describe various embodiments of thedisclosure. Accordingly, the dimensions of the various features may bearbitrarily expanded or reduced for clarity. In addition, some of thedrawings may be simplified for clarity. Thus, the drawings may notdepict all of the components of a given apparatus or all operations of aparticular method.

Information and signals described herein may be represented using any ofa variety of different technologies and techniques. For example, data,instructions, commands, information, signals, bits, symbols, and chipsthat may be referenced throughout the description may be represented byvoltages, currents, electromagnetic waves, magnetic fields or particles,optical fields or particles, or any combination thereof. Some drawingsmay illustrate signals as a single signal for clarity of presentationand description. It should be understood by a person of ordinary skillin the art that the signal may represent a bus of signals, wherein thebus may have a variety of bit widths and the disclosure may beimplemented on any number of data signals including a single datasignal.

The various illustrative logical blocks, modules, circuits, andalgorithm acts described in connection with embodiments disclosed hereinmay be implemented or performed with, for example, a general-purposeprocessor, a special-purpose processor, a Digital Signal Processor(DSP), an Application Specific Integrated Circuit (ASIC), a FieldProgrammable Gate Array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.

A processor may be a microprocessor, but in the alternative, theprocessor may be any processor, controller, microcontroller, or statemachine suitable for carrying out processes of the disclosure. Aprocessor may also be implemented as a combination of computing devices,such as a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration.

A processor may be part of a computer, which should be considered aspecial purpose computer when configured to execute instructions (e.g.,software code) for carrying out embodiments of the disclosure. Moreover,when configured according to embodiments of the disclosure, such aspecial-purpose computer improves the function of a computer because,absent the disclosure, the computer would not be able to carry out theprocesses of the disclosure. The disclosure also provides meaningfullimitations in one or more particular technical environments that gobeyond an abstract idea. For example, embodiments of the disclosureprovide improvements in the technical field of communications involvingaudibly-impaired people.

In addition, it is noted that the embodiments may be described in termsof a process that is depicted as a flowchart, a flow diagram, astructure diagram, or a block diagram. Although a flowchart may describeoperational acts as a sequential process, many of these acts can beperformed in another sequence, in parallel, or substantiallyconcurrently. In addition, the order of the acts may be re-arranged. Aprocess may correspond to a method, a function, a procedure, asubroutine, a subprogram, etc. Furthermore, the methods disclosed hereinmay be implemented in hardware, software, or both. If implemented insoftware, the functions may be stored or transmitted as one or morecomputer-readable instructions (e.g., software code) on acomputer-readable medium. Computer-readable media includes both computerstorage media and communication media including any medium thatfacilitates transfer of a computer program from one place to another.

It should be understood that any reference to an element herein using adesignation such as “first,” “second,” and so forth does not limit thequantity or order of those elements, unless such limitation isexplicitly stated. Rather, these designations may be used herein as aconvenient method of distinguishing between two or more elements orinstances of an element. Thus, a reference to first and second elementsdoes not mean that only two elements may be employed there or that thefirst element must precede the second element in some manner. Also,unless stated otherwise a set of elements may comprise one or moreelements.

Elements described herein may include multiple instances of the sameelement. These elements may be generically indicated by a numericaldesignator (e.g., 110) and specifically indicated by the numericalindicator followed by an alphabetic designator (e.g., 110A) or a numericindicator preceded by a “dash” (e.g., 110-1). For ease of following thedescription, for the most part, element number indicators begin with thenumber of the drawing on which the elements are introduced or most fullydiscussed. Thus, for example, element identifiers on a FIG. 1 will bemostly in the numerical format 1xx and elements on a FIG. 3 will bemostly in the numerical format 3xx.

Embodiments of the disclosure include apparatuses and methods foroperating a communication system. It should be noted that while theutility and application of the various embodiments of the disclosure aredescribed herein with reference to communication systems for individualshaving disabilities, the embodiments of the disclosure also include anyenvironment for a communication system to selectively operate in one ofa plurality of different modes, such as where a text mode and a tonemode may be desirable.

FIG. 1 is a simplified block diagram of a communication system 100according to an embodiment of the disclosure. The communication system100 may include a video relay service (VRS) or other communicationsystem configured to facilitate communication between users having adisability and far-end users (e.g., audibly-capable users, computerizedsystems, etc.).

The communication system 100 may include a first communication device102 and a second communication device 104 that are configured toparticipate in communication sessions through one or more networks. Thecommunication system 100 may also include a relay service 106 configuredto communicate with the first communication device 102 and the secondcommunication device 104. The first communication device 102 may includeany device configured to participate in a communication session with thesecond communication device 104, the relay service 106, or a combinationthereof.

In some embodiments, the first communication device 102 may comprise acommunication device for an individual having a disability, such as aperson having a hearing impairment, speech impairment, and/or visionimpairment. Communication devices that may be used to assist usershaving such disabilities may include a video phone device, atext-captioned device, a braille display device, keyboards, otherdevices or accessibility interfaces, and combinations thereof. The firstcommunication device 102 may include a computing device configured toexecute software directed to perform such communication capabilities.Examples of suitable computing devices may include a desktop computer, alaptop computer, a tablet computer, a personal digital assistant (PDA),a smartphone, and other computing devices.

In a typical relay system for users having disabilities, the firstcommunication device 102 may be operated by a user having a disabilityand the second communication device 104 may be operated by anaudibly-capable user. For convenience, the first communication device102 may also be referred to as the “near-end” device, while the secondcommunication device 104 may also be referred to herein as the “far-end”device. The user of the first communication device 102 may be referredto as the “near-end user,” and the user of the second communicationdevice 104 may be referred to as the “far-end user.” Of course, it isrecognized that “near-end” and “far-end” are relative terms depending onthe perspective of the particular user. Thus, the terms “near-end” and“far-end” are used as a convenient way of distinguishing between users.

The near-end user may have a disability that may make it difficult tocommunicate with the far-end user. The near-end user may be anaudibly-impaired (e.g., deaf) user, a speech-impaired (e.g., mute) user,a visually-impaired (e.g., blind) user, or have some other impairment orcombinations thereof. The far-end user may be an audibly-capable user, acomputer system (e.g., an automated system), an individual having adisability, or a combination thereof.

The relay service 106 is configured to interpret communication betweenthe first communication device 102 and the second communication device104. The operator at the relay service 106 performing the interpretationmay be referred to as a “call assistant.” Communication between thefirst communication device 102 and the relay service 106 may betypically performed through video or text communication between thenear-end user and the call assistant, while communication between therelay service 106 and the second communication device 104 may betypically performed using voice communication between the call assistantand the far-end user.

For near-end users that have vision capabilities, video data 108 may betransmitted between the first communication device 102 and the relayservice 106 to facilitate visual communication between the near-end userand the call assistant. Thus, the video data 108 may include videoinformation conveying video images of the near-end user and the callassistant. For example, the video data 108 may include video images ofthe near-end user and the call assistant. As a result, the near-end userand the call assistant may communicate with each other using signlanguage, facial expressions, lip reading, and/or other body languagebetween the near-end user and the call assistant.

In some embodiments, voice data 110 may be transmitted between the firstcommunication device 102 and the relay service 106. For example, if thenear-end user has at least some speech capabilities (e.g., users thatare hearing-impaired but have some ability to speak), voice data 110 maybe transmitted from the first communication device 102 to the relayservice 106. If the near-end user has at least some hearing-capabilities(e.g., users that are not completely deaf), voice data 110 may betransmitted from the relay service 106 to the first communication device102. In some embodiments, however, audio from the first communicationdevice 102 may be disabled such that the relay service 106 and/or thesecond communication device 104 may not be able to hear any speech orother audible noise from the first communication device 102.

In some embodiments, the first communication device 102 may include abraille display for visually-impaired users to communicate with therelay service 106. Such a braille display may include thevisually-impaired user to speak and/or type information that is sent torelay service 106 for translation to the far-end user, and receiveinformation from the relay service 106 that is converted to braille onthe first communication device 102 for the near-end user to understand.

The relay service 106 and the second communication device 104 may beconfigured to communicate voice data 110 with each other to facilitateaudible communication between the call assistant and the far-end user(e.g., a live person, a computer system, etc.). The voice data 110 mayinclude audible information shared between the relay service 106 and thesecond communication device 104. For example, the voice data 110 mayinclude audible speech of the call assistant and the far-end user. As aresult, the communication system 100 may facilitate translatedcommunication for near-end users with others regardless of whether thefar-end user is able to communicate using the mode of communication forthe near-end user. In some embodiments, the voice data 110 transmittedto the second communication device 104 may also include audible speechof the near-end user if the first communication device 102 is configuredto transmit audio, and such audio is desired to be heard by the far-enduser. Voice data 110 generated by the far-end user is not limited tolive voice from a live person. For example, the far-end user may includean automated telephone menu system that transmits voice data 110 thatincludes synthesized voice, recorded voice, or other audible informationintended to convey information (e.g., interactive voice response (IVR)).

In some embodiments, the second communication device 104 may beconfigured to also send and receive video communications. For example,both the first communication device 102 and the second communicationdevice 104 may be video phones of a VRS communication system. In suchembodiments, the second communication device 104 may be configured toreceive the video data 108 including the video images of the near-enduser, the call assistant, or a combination thereof. The secondcommunication device 104 may be configured to receive the video data 108from the first communication device 102, the relay service 106, orcombinations thereof. Receipt of the video data 108 by the secondcommunication device 104 may enable the far-end user to observenon-verbal communication from the near-end user. Thus, the communicationsystem 100 may also be configured to facilitate communication betweennear-end users.

At times, it may be desirable for the far-end user, the call assistant,or combinations thereof, to receive additional communication may bedesirable for the near-end user to transmit as character data 107 thatis not voice data 110 or video data 108. By way of non-limiting example,where the far-end user includes IVR technology, it may be desirable forthe far-end user to receive tones 111 (e.g., DTMF tones). In suchsituations, the call assistant may translate (e.g., via sign language,text captioning, etc.) menu options from the far-end user for thenear-end user. As discussed above, conventional communication systemsmay require the near-end user to respond by using sign language via thevideo data 108, whereupon the call assistant may use a numeric keypad atthe relay service 106 to send tones 111 to the second communicationdevice 104. Such a system may result in errors, inefficiency, and/or anundesirable user experience because of multiple steps in thetranslation. Also by way of non-limiting example, the far-end user maybe a live person that requests character-intensive information, such aslengthy numbers (e.g., phone numbers, social security numbers, etc.),passwords, or codes. Conventional communication systems may require thenear-end user to respond by using sign language via the video data 108,whereupon the call assistant may translate the sign language to speechfor the far-end user via the voice data 110. Such a system may alsoresult in errors, inefficiency, and/or an undesirable user experience.

Embodiments of the disclosure enable the near-end user at the firstcommunication device 102 to convey character data 107 to the relayservice 106 and/or the second communication device 104. Such characterdata 107 may be different than the video data 108 and the voice data 110transmitted by the communication system 100. As an example, the firstcommunication device 102 may include an input device (not shown inFIG. 1) configured to generate the character data 107 responsive to auser input. The input device may include a numeric keypad, a keyboard, atouchscreen, a remote control, a mouse, other input device, orcombinations thereof. The character data 107 may be transmitted as tones111, text signals 109, or other formats for conveying character-basedinformation.

Depending on the capabilities of the far-end user, it may be desirablefor the far-end user to receive certain types of character-basedinformation, but not others. The communication system 100 may beconfigured to selectively operate in one of a plurality of differentmodes that may determine how the character data 107 is conveyed to therelay service 106 and/or the second communication device 104. Forexample, the communication system 100 may be configured to selectivelyoperate in at least one of a tone mode and a text mode. In someembodiments, the tone mode may comprise a DTMF mode, and the text modemay comprise an RTT mode. For convenience, the tone mode and the textmode may be referred to herein as the DTMF mode and the RTT mode,respectively.

The term “tone mode,” as used herein, refers to a functional mode inwhich at least one audible tone (e.g., tone 111) is received by thesecond communication device 104 responsive to the user input from thefirst communication device 102 regardless of the format of the characterdata 107 generated by the first communication device 102. The audibletone 111 corresponds to the information input by the near-end user whileconveying the character data 107.

For example, the first communication device 102 may include an inputdevice having a numeric keypad with keys corresponding to the numbers 0through 9, “*” and “#.” If the near-end user selects (e.g., presses) oneof the keys, the first communication device 102 may generate thecharacter data 107 corresponding to the key selected by the near-enduser. The character data 107 generated by the first communication device102 may be formatted as a tone 111 or a text signal 109. In tone mode,however, the character data 107 that is received by the secondcommunication device 104 may be formatted as a tone 111 regardless ofhow the character data 107 was first generated.

In some embodiments, the character data 107 may initially be generatedby the first communication device 102 as a tone 111. In such anembodiment, the tone 111 may be transmitted from the first communicationdevice 102 to the second communication device 104 directly, if such adirect connection exists. In some embodiments, the tone 111 may betransmitted from the first communication device 102 to the relay service106, which may be configured to route the tone 111 to the secondcommunication device 104.

In some embodiments, the character data 107 may initially be generatedby the first communication device 102 as a text signal 109. In such anembodiment, the text signal 109 may be transmitted from the firstcommunication device 102 to the second communication device 104. Therelay service 106 may receive the text signal 109 and generate theappropriate tone 111 corresponding to the information being conveyed.The relay service 106 may then transmit the tone 111 to the secondcommunication device 104. As a result, during tone mode the relayservice 106 may transmit the tone 111 to the second communication device104 regardless of the format that the character data 107 was received bythe relay service 106.

By way of non-limiting example, the tone 111 may include two differentfrequencies depending on the input key selected by the near-end user, asillustrated in Table 1.

TABLE 1 1209 Hz. 1336 Hz. 1477 Hz. 697 Hz. 1 2 3 770 Hz. 4 5 6 852 Hz. 78 9 941 Hz. * 0 #For example, if the “5” key is selected by the near-end user, theappropriate tone 111 may be generated by adding two sinusoidal signals,one with a frequency of 1336 Hz., and the other with a frequency of 770Hz. The tones 111 may be received by the second communication device104, and analysis of the received tones may reveal that the “5” key wasselected.

As briefly mentioned above, the communication system 100 may also beconfigured to operate in a text mode. The term “text mode,” as usedherein, refers to a functional mode in which at least one text signal(e.g., text signal 109) is displayed to the call assistant and/or thefar-end user regardless of the format of the character data 107generated by the first communication device 102. If the text isdisplayed to the call assistant, the call assistant may then speak themessage from the text to the far-end user at the second communicationdevice 104.

For example, the near-end user may select a user input at the firstcommunication device 102 to convey a message to the far-end user usingcharacter data 107. The character data 107 may be transmitted as a textsignal 109, a tone 111, or some other format. In text mode, characterdata 107 may be displayed as text on an electronic display within viewof the call assistant regardless of the format that the character data107 was generated by the first communication device 102. For example, insome embodiments the character data 107 may be received by the relayservice 106 as a text signal 109, whereupon the character data 107 maybe displayed as text on the call assistant's electronic display withlittle, to no, additional conversion of the textual data. In someembodiments, however, the character data 107 may be received by therelay service 106 as a tone 111 that the relay service 106 converts tothe corresponding text displayed on the call assistant's electronicdisplay. The call assistant may then read the text and speak with thefar-end user via the voice data 110. Thus, the text mode may includeaudio being transmitted to the second communication device 104 to conveythe message from the character data 107.

For example, the far-end user may request information from the near-enduser (via the relay service 106) that may be desirable to transmitthrough character data 107 generated by user inputs to an input devicerather than sign language. Examples of such data may include numericalinformation such as a social security number, an account number, a phonenumber, etc. The call assistant may listen to the request from thefar-end user via voice data 110 between the second communication device104 and the relay service 106, and communicate the request using signlanguage to the near-end user via video data 108 between the relayservice 106 and the first communication device 102. The near-end usermay respond to the request by entering the appropriate characterinformation into the input device. The call assistant may receive thecharacter data 107 on the electronic display at the relay service 106via text signal 109, and speak the character data 107 to the far-enduser via voice data 110. In some embodiments, the second communicationdevice 104 may be configured to accept a text signal 109. In such anembodiment, the second communication device 104 may be configured toreceive the text signal 109 in addition to, or in the alternative of,the call assistant speaking the character information to the far-enduser during text mode.

Thus, the communication system 100 may be configured to switch betweenmodes that convey a message from character data 107 (e.g., digitcommands from the near-end user) either as tones, audio, or textdepending on the capabilities of the far-end user. It may be desirableto operate the relay service 106 in the text mode while a far-end useris a live person, as opposed to in the tone mode. For example, theaudible tones associated with the tone mode may be distracting orunpleasant to a live person. It is also unlikely for a live person to beable to interpret a useful meaning from the audible tones of tone modesignals. Therefore, during text mode, even if the character data 107 isreceived by the relay service 106 as a tone 111, the information may beconveyed to the far-end user without the relay service 106 passing onthe audible tones 111 to the second communication device 104. Rather,the information may be displayed as text on the electronic display ofthe relay service 106, and the call assistant may speak the message asaudio that is transmitted to the second communication device with thevoice data 110. In some embodiments, the second communication device 104may be configured to receive text data 109 such that relay service 106may convey the message as text data 109 to the second communicationdevice 104.

It may be desirable to operate the relay service 106 in the tone mode ifthe far-end user is configured to accept DTMF signals. In general,accepting DTMF signals means that the far-end user is not a live person.For example, the second communication device 104 may be configured totransmit one of recorded speech signals and synthesized speech signals(e.g., voice data 110) to the relay service 106, the first communicationdevice 102, or combinations thereof during a communication session.Recorded speech and synthesized speech messages often requestinteraction through the selection of keys that result in audible tones111 being sent to the second communication device 104. By way ofnon-limiting example, automatic answering services, over the telephoneautomatic bill payment, banking, surveys, and other services oftenrequest a user to make selections that generate audible tones 111. As aresult, the communication system 100 may be configured to operate in thetone mode responsive to a determination that a live person is notoperating the second communication device 104.

The communication system 100 may be configured to operate in the textmode responsive to a determination that a live person is operating thesecond communication device 104. The determination of whether a far-enduser is a live person participating in the communication session may bemade manually, automatically, or a combination thereof.

A manual determination may be made by the call assistant distinguishinglive speech from recorded speech and/or synthesized speech. The callassistant may be prompted to provide a result of the determination. Forexample, the call assistant may make a selection on an input device atthe relay service 106 to indicate which mode is desired.

An automatic determination may be made by analyzing speech signals fromthe second communication device 104. The automatic determination may bemade by the relay service 106, the first communication device 102, orcombinations thereof. For example, the voice data 110 received from thesecond communication device 104 may be analyzed to determine if arecording, synthesized speech, or a live person is the source of thevoice data 110. Different characteristics of the voice data 110 may beanalyzed, such as audio quality, cadence, or other factors. For example,a system and method for detecting recorded speech is described in U.S.Pat. No. 6,480,825 to Sharma et al., and entitled “System and Method forDetecting a Recorded Voice,” the disclosure of which is herebyincorporated herein in its entirety by this reference. As an example,Sharma et al. disclose training a pattern classifier, such as a NeuralTree Network, to distinguish between live and recorded speech. Inaddition, an apparatus and method for detecting errors in synthesizedspeech is described in U.S. Pat. No. 6,993,488 to Vainio et al., andentitled “Audible Error Detector and Controller Utilizing ChannelQuality Data and Iterative Synthesis,” the disclosure of which is herebyincorporated herein in its entirety by this reference. In particular,Vainio et al. disclose that characteristics atypical of human speech maybe detected in synthesized speech.

In some embodiments, both a manual determination and an automaticdetermination of whether a live person is operating the secondcommunication device 104 may be made. For example, an automaticdetermination may be made, and the call assistant may be prompted toconfirm the determination. Also by way of example, an automaticdetermination may be attempted, and the call assistant may be promptedto make a manual determination if the automatic determination fails tomake an adequate determination. In addition, an automatic determinationmay be made, but the call assistant may be able to manually override theautomatic determination and provide a manual determination.

In addition, the communication system 100 may change modes during thesame communication session as conditions may change. For example, acommunication session may initially be answered by an automated menusystem that eventually leads to a live operator as the far-end user.Some communication sessions may include a live operator transferring thecall to a computerized system. At such transitions, a new mode may bedetected by the communication system 100 to change the operating modethereof.

In some embodiments, the relay service 106 may be configured to causethe communication system 100 to operate in the appropriate one of thetone mode and the text mode. In these embodiments, the phrases“operating the communication system 100 in the tone mode” and “operatingthe communication system 100 in the text mode” refer to the relayservice 106 causing the communication system 100 to operate in the tonemode, and the text mode, respectively. In some embodiments, this modeselection involves the relay service 106 receiving the character data107 from the first communication device 102 in a first format, andconverting the character data 107 to a second format, as needed for thecurrent mode. In some embodiments, however, the relay service 106 mayinform the first communication device 102 of the desired mode. The firstcommunication device 102 may then generate the character data 107 in thedesired format for the current mode so that conversion by the relayservice 106 may not be necessary. In either embodiment, the near-enduser may not need to be aware of the mode or require using a differentuser interface depending on the mode.

In some embodiments, the first communication device 102 may beconfigured to cause the communication system 100 to operate in theappropriate one of the tone mode and the text mode. In theseembodiments, the phrases “operating the communication system 100 in thetone mode” and “operating the communication system 100 in the text mode”refer to the first communication device 102 causing the communicationsystem 100 to operate in the tone mode, and the text mode, respectively.In some embodiments, that may involve the first communication device 102determining which operating mode should be employed by determiningwhether the far-end user is not a live person. Such a determinationbeing made by the first communication device 102 may be an automaticdetermination by analyzing the voice data 110 generated by the secondcommunication device 104 if such voice data 110 is received by the firstcommunication device 102. In other words, some embodiments may includethe analysis of the far-end voice data 110 being executed by the firstcommunication device 102. As the near-end user may have significantlydiminished hearing capabilities, a manual determination by the firstcommunication device 102 may not be desirable.

Regardless of which one of the relay service 106 and the firstcommunication device 102 is configured to cause the communication system100 to operate in the appropriate one of the tone mode and the textmode, the end result is similar. For example, in either situation, thetone 111 may be received by the second communication device 104 duringtone mode regardless of how the first communication device 102 generatedthe character data 107. Similarly, in either situation, an indicator ofthe text signal 109 may be displayed by the relay service 106 and/or thesecond communication device 104 during text mode.

The first communication device 102, the second communication device 104,and the relay service 106 may be configured to transmit the data signals108, 109, 110, 111 over one or more networks. It should be recognizedthat even though the data signals 108, 109, 110, 111 are illustrated inthe figures as individual lines, that does not mean that the datasignals 108, 109, 110, 111 are transmitted separately. For example, someof the data signals 108, 109, 110, 111 may be transmitted together.

The networks used for conveying the data signals 108, 109, 110, 111 mayinclude an internet protocol (IP) network. The networks may also includeother networks, such as, for example, public switched telephone networks(PSTNs). The networks may include a wide area network (WAN), a localarea network (LAN), a personal area network (PAN), and combinationsthereof. In some embodiments, the networks may include a cloud network.The networks may be configured to facilitate wireless communications,communications through cables, and combinations thereof. Somenon-limiting examples of suitable wireless communications may include“WiFi,” Bluetooth, and mobile wireless networks. Some non-limitingexamples of suitable cables include fiber-optic cables, coaxial cables,traditional telephone cables, and Ethernet cables.

FIG. 2 is a simplified block diagram of the first communication device102 of FIG. 1. The first communication device 102 may include aprocessor 220 operably coupled with a camera 222, an electronic display224, one or more communication elements 228, and a memory device 230.The first communication device 102 may be configured to communicatevideo data 108 from the camera 222 to the relay service 106 (FIG. 1)through the communication elements 228. The video data 108 captured bythe camera 222 may include sign language communication originated by thenear-end user. The first communication device 102 may also be configuredto receive video data 108 from the relay service 106 through thecommunication elements 228 to be displayed by the electronic display224. The video data 108 received by the first communication device 102may include sign language communication originated by the call assistantat the relay service 106. The processor 220 may coordinate thecommunication between the various devices as well as executeinstructions stored in computer-readable media of the memory device 230.The memory device 230 may include volatile and non-volatile memorystorage for the first communication device 102.

In some embodiments, the first communication device 102 may beconfigured to cause the communication system 100 to operate in aplurality of different modes of operation. In some embodiments, theprocessor 220 and the memory device 230 may form a control circuit thatis configured to determine which mode of operation the communicationsystem 100 should operate in. As discussed in detail above, the firstcommunication device 102 may be configured to cause the communicationsystem 100 to operate in a text mode responsive to a determination thatthe far-end user is a live person operating the second communicationdevice 104 that is participating in a communication session with thefirst communication device 102. The first communication device 102 mayalso be configured to cause the communication system 100 to operate inthe tone mode responsive to a determination that the far-end user is nota live person operating the second communication device 104. Rather, thefar-end user may be an automated computer system that employssynthesized voice and/or recorded voice.

In some embodiments, the first communication device 102 may beconfigured to automatically determine whether a live person is operatingthe second communication device 104. For example, the processor 220 maybe configured to analyze a speech signal from the second communicationdevice 104 to automatically make the determination. The communicationelements 228 may be configured to deliver the speech signal as the voicedata 110 from the second communication device 104 to the processor 220.

In operation, an input device 226 may receive an input from the near-enduser. For example, the input device 226 may include a numeric keypad, akeyboard, a touchscreen, a remote control, a mouse, other input devices,or combinations thereof. For example, during a communication session,the near-end user may desire to enter character data 107 to transmit tothe relay service 106, the second communication device 104, or acombination thereof.

For example, the far-end user may include an automated phone menu systemthat is requesting information from the user. Thus, the communicationsystem 100 may operate in the tone mode. Rather than using sign languageto convey this information to the call assistant, the near-end user mayenter the information into the input device 226 to generate thecharacter data 107 corresponding to the information entered into theinput device 226. The character data 107 may be represented by the textsignal 109, the tones 111, or both. The text signal 109 may includetextual data, or other data corresponding to the information input bythe near-end user. In some embodiments, the text signal 109 is sent tothe relay service 106, which may generate the tone 111 that istransmitted to the second communication device 104. Thus, theinformation entered into the input device 226 may be conveyed to thesecond communication device 104 directly without the call assistantneeding to translate such information from interpreting sign languageand then entering the information for the near-end user. As a result,accuracy, efficiency, and/or user experience may be improved with regardto conveying this information to the far-end user. In some embodiments,embodiments, the character data 107 generated by the first communicationdevice 102 as a tone 111 may be sent directly, or routed through therelay service 106, to the second communication device 104.

In some embodiments, the near-end user may desire to transmit characterdata 107 to the relay service 106 even if the far-end user is a liveperson and/or does not accept tones 111. Thus, the communication system100 may operate in the text mode. For example, the far-end user mayrequest information that the near-end user would prefer to enter throughan input device 226 rather than using sign language to communicate withthe call assistant. The character data 107 may be transmitted to therelay service 106, which may display the character data 107 as text tothe call assistant regardless of the format of the character data 107.In the text mode, if the text is displayed to the call assistant, thecall assistant may speak the message to the far-end user at the secondcommunication device 104.

In some embodiments, the relay service 106 may be configured todistinguish whether the second communication device 104 may beconfigured to receive the text signal 109 or whether voice data 110would be more desirable for conveying the information associated withthe text signal 109 as will be discussed more fully with respect to FIG.3. As for the near-end user at the first communication device 102,knowing the actual way the relay service 106 conveys the information tothe second communication device 104 may not be necessary.

In addition, as the determination of the appropriate operational mode ismade automatically by the relay service 106, or manually by the callassistant, the near-end user need not know what operational mode thecommunication system 100 is operating in. In fact, the near-end userneed not even know of the existence of the different operating modes ofthe communication system 100. Thus, embodiments of the disclosureprovide for simplified and effective operation of the firstcommunication device 102 from the perspective of the near-end user.

FIG. 3 is a simplified block diagram of a computing device 300 for thecall assistant of the relay service 106 of FIG. 1. The computing device300 may include a processor 320 operably coupled with a camera 322, anelectronic display 324, one or more communication elements 328, and amemory device 330. The computing device 300 may be configured tocommunicate video data 108 from the camera 322 to the firstcommunication device 102 (FIG. 1) through the communication elements328. The video data 108 captured by the camera 322 may include signlanguage communication originated by the call assistant. The computingdevice 300 may also be configured to receive video data 108 from thefirst communication device 102 through the communication elements 328 tobe displayed by the electronic display 324. The video data 108 receivedby the first communication device 102 may include sign languagecommunication originated by the near-end user at the first communicationdevice 102. The computing device 300 may also be configured to send andreceive voice data 110 between the relay service 106 and the secondcommunication device 104. As the communication sessions may includespeech communications, the computing device 300 may include a microphoneand a speaker (not shown) to enable such speech communications.

The processor 320 may coordinate the communication between the variousdevices as well as execute instructions stored in the memory device 330.The memory device 330 may include volatile and non-volatile memorystorage for the computing device 300. In some embodiments, the relayservice 106 may be configured to cause the communication system 100 tooperate in a plurality of different modes of operation. The processor320 and the memory device 330 may form a control circuit that isconfigured to determine which mode of operation the communication system100 should operate in.

In some embodiments, the relay service 106 may be configured to causethe communication system 100 to operate in a text mode responsive to adetermination that the far-end user is a live person operating thesecond communication device 104 that is participating in a communicationsession with the first communication device 102. The relay service mayalso be configured to cause the communication system 100 to operate inthe tone mode responsive to a determination that the far-end user is nota live person operating the second communication device 104. Rather, thefar-end user may be an automated computer system that employssynthesized voice and/or recorded voice.

In some embodiments the computing device 300 may be configured to detecta manual determination of whether a live person is operating the secondcommunication device 104. For example, the input device 326 may beconfigured to receive a user selection indicating a result of the manualdetermination. By way of non-limiting example, the input device 326 maybe configured to receive a user selection indicating whether a liveperson is operating the second communication device 104. Also by way ofnon-limiting example, the input device 326 may be configured to receivea user selection indicating whether the relay service 106 should operatein the text mode or the tone mode. The input device 326 may beconfigured to present selectable options directed to indicating whethera live person is operating the second communication device 104, whichmode the relay service 106 should operate in, and combinations thereof.

In some embodiments, the computing device 300 may be configured toautomatically determine whether a live person is operating the secondcommunication device 104. For example, the processor 320 may beconfigured to analyze a speech signal from the second communicationdevice 104 to automatically make the determination. The communicationelements 328 may be configured to deliver the speech signal from thesecond communication device 104 to the processor 320.

In some embodiments, the processor 320 may be configured to enable thecall assistant to manually override the automatic determination. By wayof non-limiting example, the processor 320 may be configured to detect auser selection of a manual override option presented to the callassistant by the input device 326. In addition, in some embodiments, theprocessor 320 may be configured to enable the call assistant to confirmthe automatic determination. By way of non-limiting example, theprocessor 320 may be configured to cause the input device 326 to promptthe user to confirm a result of the automatic determination.

In some embodiments, the processor 320 may be configured to cause thecommunication system 100 to switch from operating in one of the textmode and the tone mode to the other of the text mode and the tone moderesponsive to a subsequent different determination of whether a liveperson is operating the second communication device 104. For example, amanual determination that a live person is operating the secondcommunication device 104 may be made, and the processor 320 may causethe communication system 100 to operate in the text mode. The callassistant that made the manual determination may later decide that thesecond communication device 104 is transmitting speech signalscorresponding to recorded speech. The call assistant may make thesubsequent determination that no live person is operating the secondcommunication device 104. The processor 320 may cause the communicationsystem 100 to operate in the tone mode responsive to the subsequentdetermination that no live person is operating the second communicationdevice 104.

In some embodiments, automatic determinations of whether a live personis operating the second communication device 104 may be madeperiodically throughout a communication session. For example, one of amanual and automatic determination may be made near a beginning of acommunication session, and periodic automatic determinations may besubsequently made. Regardless of whether the first communication device102 or the relay service 106 makes the automatic determination, thecommunication system may operate in the one of the text mode and thetone mode that corresponds to the most recent determination.

In some embodiments, one of the first communication device 102 and therelay service 106 may be configured to monitor communication signalsfrom the second communication device 104. The one of the firstcommunication device 102 and the relay service 106 may also beconfigured to cause subsequent manual or automatic determinations to bemade responsive to changes in one or more characteristics of thecommunication signals from the second communication device 104. Forexample, a sudden increase or decrease in levels of background noise maybe indicative of a change between recorded and live speech. Otherchanges in characteristics that may be significant include changes inchannel characteristics, pitch of speech, loudness of speech, signalenergy, other characteristics, and combinations thereof.

In some embodiments, the one of the first communication device 102 andthe relay service 106 may be configured to cause the communicationsystem 100 to default to operating in one of the text mode and the tonemode before the determination of whether a live person is operating thesecond communication device 104. Also, the one of the firstcommunication device 102 and the relay service 106 may be configured tocause the communication system to default to operating in one of thetext mode and the tone mode responsive to a failure to make anacceptable determination. For example, the one of the firstcommunication device 102 and the relay service 106 may be configured tocause the communication system 100 to operate in the text mode unlessthere is a determination that the second communication device 104 is notoperated by a live person.

FIG. 4 is a flowchart 400 illustrating a method of operating acommunication system 100 according to an embodiment of the disclosure.At operation 410, the relay service 106 may receive incoming characterdata 107 in the form of an incoming text signal 109 from the firstcommunication device 102. The text signal 109 may correspond toinformation desired to be conveyed by a near-end user to the secondcommunication device 104. At operation 420, the relay service 106 maydetermine which operational mode the relay service 106 is operating in.The relay service 106 may determine an operational mode through a manualinput of the call assistant (e.g., through a user interface), through anautomatic determination by analyzing the audio data being received bythe second communication device 104, through an automatic determinationby the first communication device 102, or combinations thereof.

If the communication system 100 is in a tone mode (e.g., the far-enduser is a computerized system that accepts DTMF tones), at operation 430the relay service 106 may send tones 111 corresponding to the characterdata 107 to the second communication device 104 regardless of the formatof the character data 107. In some embodiments, the relay service 106may simply route the character data 107 to the second communicationdevice 104 if the character data 107 is already configured as tones 111.In some embodiments, the relay service 106 may convert the characterdata 107 received as the text signal 109 to tones 111 that aretransmitted to the second communication device 104.

If the communication system 100 is in the text mode (e.g., the far-enduser does not accept DTMF tones, such as being a live person), atoperation 440 the relay service 106 may display the character data 107as real-time text on the electronic display 324 of the call assistantregardless of the format of the character data 107. For example, therelay service 106 may simply receive and display the character data astext with little, to no conversion, if the character data 107 is alreadya text signal 109. In some embodiments, the relay service 106 mayconvert the character data 107 to text if the character data 107received in another format (e.g., tones 111).

At operation 450, the relay service 106 may convey the message to thesecond communication device 104, such as by the call assistant speakingthe text displayed on the electronic display of the relay service 106 tothe far-end user as voice data 110. In some embodiments, the secondcommunication device 104 may be configured to accept textual signals. Asa result, the relay service 106 may also be configured to transmit theinformation from the character data 107 to the second communicationdevice 104 as textual data in addition to, or instead of, displaying thetextual data to the call assistant to speak to the far-end user.

FIG. 5 is a flowchart 500 illustrating a method of operating acommunication system 100 according to an embodiment of the disclosure.At operation 510, the first communication device 102 may receive a userinput corresponding to a desired message to convey to the secondcommunication device 104. At operation 520, the first communicationdevice 102 may determine which operational mode the communication system100 is operating in. For example, the first communication device 102 mayhave received an indication from the relay service 106 a desired modebased on a determination by the relay service 106 that the far-end useris one of a live person or an automated system. In some embodiments, thefirst communication device 102 may perform at least some of theautomated analysis of the far-end speech to automatically determinewithin the first communication device 102 which mode the communicationsystem 100 should be operating in. The first communication device 102may inform other devices of the communication system 100 of selected themode.

If the communication system 100 is in a tone mode, at operation 530, thefirst communication device 102 may transmit the character data 107 astones 111. If the communication system 100 is in the text mode, atoperation 540, the first communication device 102 may transmit thecharacter data 107 as a text signal 109. It should be noted that theembodiment of FIG. 5 describes a situation, in which the firstcommunication device 102 may be aware of the operational mode (whetherby its own detection or being informed of such). As discussed previouslyherein, it is contemplated that the first communication device 102 maynot be aware of the operational mode. As such, the first communicationdevice 102 may transmit the character data 107 to one or both of therelay service 106 and the second communication device 104 in a formatthat does not match how the relay service 106 and/or the secondcommunication device 104 desires for the particular operational mode.Thus, the relay service 106 may convert the character data 107 to theappropriate format for the current operational mode.

FIG. 6 is a simplified block diagram of a communication system 600according to embodiments of the disclosure. The communication system 600may include video endpoints 602-1 and 602-2 (sometimes referred toherein generally individually as “video endpoint” 602 and together as“video endpoints” 602), similar to the first communication device 102 ofFIGS. 1 and 2. The video endpoints 602 may be configured to communicatevideo 608 and text 609 to each other, similar to the first communicationdevice 102 communicating video 108 and text 109 to the relay service106, as discussed above with reference to FIG. 1. Accordingly, the videoendpoints 602 may include communication devices for audibly-impairedusers (e.g., video phones).

Each of the video endpoints 602 may include video equipment 646 (e.g.,including the camera 222 and the electronic display 224 of FIG. 2), acontrol circuit 644 and a data storage 640 (also referred to hereinsimply as “storage” 640). The storage 640 may include a database 642 forstoring strings of text that may be selected and sent to another of thevideo endpoints 602 as text 609 without a user of the video endpoint 602having to type out the strings of text during a video call with theother video endpoint 602. In other words, rather than input charactersone at a time using an input device 226 (FIG. 2) to send text 609 to theother video endpoint 602, the user may instead select one of the textstrings stored in the database 642, and the video endpoint 602 maytransmit the selected text string to the other video endpoint 602. Theability to select and transmit text strings without having to providecharacter inputs one by one may be especially useful where the inputdevice 226 includes a cumbersome character selection interface. Forexample, a video phone may include a remote control for providingcharacter inputs, which may provide for a cumbersome character selectioninterface relative to a character selection interface such as akeyboard.

The database 642 may store text strings that correspond to textualinformation that the user frequently provides during video calls. By wayof non-limiting example, the database 642 may store text stringsconveying account numbers, mailing addresses, email addresses, telephonenumbers, prescription information, payment information, drivinginstructions, restaurant orders, and other information that may beconveyed by text during a video call.

As a specific, non-limiting example, the database 642 of the videoendpoint 602-1 may store a text string including a residential addressof the user of the video endpoint 602-1. The user of the video endpoint602-1 may place a video call to another user of another video endpoint602-2 to tell the other user where the user's place of residence islocated. The other user may answer the user's call, and videocommunications may be established between the video endpoints 602 (e.g.,video 608 is exchanged between the video endpoints 602). The other usermay ask the user for the user's residential address. Rather thantediously signing the residential address to the other user (e.g., usingAmerican Sign Language (ASL)), or inputting characters one by one usinga cumbersome text interface, the user may instead issue commands to thevideo endpoint 602-1 to send the text string including the residentialaddress stored in the database 642 to the other video endpoint 602-2. Inresponse, the video endpoint 602-1 may send the text string includingthe user's residential address to the other video endpoint 602-2, whichmay display the text string to the other user.

In some embodiments, the text strings may be stored in a database 642 ata remote server 660 (e.g., a computing device) instead of, or inaddition to, the databases 642 of the video endpoints 602. Accordingly,the server 660 may include a storage 662 for storing the database 642.The video endpoints 602 may communicate with the server 660 to accessthe database 642 (e.g., using communication elements 228 discussed abovewith reference to FIG. 2). By way of non-limiting example, the videoendpoints 602 may retrieve text strings from the database 642 of theserver 660, and transmit the retrieved text strings to another videoendpoint 602 for display by the other video endpoint 602. Also by way ofnon-limiting example, the video endpoints 602 may instruct the server660 to transmit the text strings directly to another video endpoint 602for display by the other video endpoint 602.

The server 660 may enable sharing of the text strings in the database642 among several video endpoints 602 that are operated by the sameuser. For example, the user may own several video endpoints 602,including a home video phone, a work video phone, and a mobile computingdevice modified with software to operate as a video endpoint 602. Theuser may change the database 642 using one of the video endpoints 602 byadding text strings to, removing text strings from, or modifying textstrings in the database 642. The changed database 642 may be availableto any of the other video endpoints 602 operated by the user.Accordingly, the user would not need to enter the same text strings intoeach of the separate video endpoints 602 operated by the user.

It should be noted that security measures may be taken to protectinformation in the database 642 of the server 660 as the text stringsmay include information that is confidential or sensitive (e.g., creditcard numbers, social security numbers, account numbers, contactinformation, etc.). For example, encryption, firewalls, other securitymeasures, and combinations thereof may be taken to protect theinformation in the database 642 while being stored, while beingcommunicated, or a combination thereof. In some embodiments, measuressimilar to those required for credit card transaction systems by thePayment Card Industry Security Standards Council® may be taken toprotect the information in the database 642 of the server 660.

In some embodiments, one of the video endpoints 602 may be part of arelay service 606 configured to provide translation services to enableaudibly-impaired users of video endpoints 602 to communicate with anaudibly-capable user of an audio endpoint 604-2 (e.g., traditional orcellular telephones). For example, the relay service 606 may include thevideo endpoint 602-2 to enable video communications between a callassistant at the relay service 606 and a user of video endpoint 602-1.The relay service 606 may also include an audio endpoint 604-1configured to enable the translator to communicate with anaudibly-capable user of another audio endpoint 604-2 using audio 610. Itshould be noted that, although the video endpoint 602-2 and the audioendpoint 604-1 of the relay service 606 are illustrated separately, thevideo endpoint 602-2 and the audio endpoint 604-1 may be integratedtogether, separately, or a combination thereof.

In embodiments where the communication system 600 includes a relayservice 606, the audibly-impaired user may select and send one of thetext strings stored in the database 642 of the video endpoint 602-1 tothe video endpoint 602-2 of the relay service 606. The selected textstring may be displayed to the call assistant at the relay service 606,and the call assistant may speak the contents of the text to theaudibly-capable user of the audio endpoint 604-2.

As a specific, non-limiting example, an audibly-impaired user of thevideo endpoint 602-1 may have a description of a favorite pizza that theaudibly-impaired user frequently orders from a pizza restaurant storedin the database 642 of the video endpoint 602-1. The audibly-impaireduser may call the pizza restaurant with the assistance of the relayservice 606. A video connection may be established between the videoendpoint 602-2 of the relay service 606 and the video endpoint 602-1.Also, an audio connection may be established between the audio endpoint604-1 of the relay service 606 and the audio endpoint 604-2 (e.g., atelephone) at the pizza restaurant. Instead of signing the descriptionof the pizza to the translator at the relay service 606, theaudibly-impaired user may input commands to the video endpoint 602-1 totransmit the text string having the description of the favorite pizza tothe video endpoint 602-2 of the relay service 606. The text stringhaving the description of the favorite pizza may be displayed to thecall assistant at the relay service 606. The call assistant may speakthe description of the favorite pizza to an audibly-capable worker atthe pizza restaurant, and the audio endpoint 604-1 of the relay service606 may transmit the audio description to the audio endpoint 604-2 atthe pizza restaurant via the audio 610. Accordingly, theaudibly-impaired user may make a pizza order without having to sign thedescription of the pizza to the translator, and without having to inputtext characters of the description of the pizza one-by-one during thecall.

In some embodiments, the video endpoint 602 may be configured to enablethe user to select a text string for transmitting to other videoendpoint 602, and modify the text string. Such modification may occureither before or after the text string has been transmitted. In theexample of the pizza order, perhaps the audibly-impaired user desires toorder the favorite pizza, but with an additional topping not listed inthe text string. The audibly-impaired user would thus be enabled toselect the text string describing the favorite pizza for transmitting tothe other video endpoint 602, and then add text describing the extratopping.

In some embodiments, the video endpoints 602 may transmit text stringsusing a real-time text protocol. In such embodiments, a text string maybe selected and the text string may be transmitted to and displayed bythe other video endpoint 602 in at least substantially real-time. Theuser may then modify the text string, and the modifications may bedisplayed in substantially real-time by the other video endpoint 602 tothe other user as the modifications are made.

In some embodiments, the video endpoint 602 may provide the user anopportunity to modify text strings before transmittal to the other videoendpoint 602. For example, a user may select a text string, modify thetext string, and then provide the video endpoint 602 instructions totransmit the modified text string to the other video endpoint 602. Thevideo endpoint 602 may transmit the modified text string, and the othervideo endpoint 602 may receive and display the modified text string.

The video endpoints 602 may also be configured to enable the user of thevideo endpoint 602 that transmitted a text string to delete or removethe text string from the display of the other video endpoint 602 thatreceived the text string.

In some embodiments, after a user transmits a text string from a videoendpoint 602 to another video endpoint 602, another user at the othervideo endpoint 602 may modify the text string, which may be displayedboth on the other video endpoint 602 and the video endpoint 602. Eitheror both of the users may save the modified text string to theirrespective databases 642.

In some embodiments, the video endpoint 602 may be configured to embedthe text to be transmitted to the other video endpoint 602 into thevideo 608 sent to the other video endpoint 602. In such embodiments, thetext may be displayed by the other video endpoint 602 as part of thevideo 608 transmitted to the other video endpoint 602. Accordingly, insuch embodiments, text 609 separate from the video 608 may not betransmitted between video endpoints 602.

In some embodiments, the video endpoint 602 may be capable of bothsending video 608 with the text embedded therein, and sending text 609separate from the video 608. In some embodiments, the video endpoint 602may be configured to determine whether or not it is communicating withanother video endpoint that is capable of receiving text 609 that isseparate from the video 608. In such embodiments, the video endpoint 602may transmit text 609 that is separate from the video 608 if the videoendpoint 602 determines that the other video endpoint is capable ofreceiving separate text 609 and video 608. Also, the video endpoint 602may transmit video 608 with the text embedded into the video 608 if thevideo endpoint 602 determines that the other video endpoint is notcapable of receiving separate text 609 and video 608. As a result, thevideo endpoint 602 may be capable of conveying text, and text strings,to the other user of the other video endpoint 602 even if the othervideo endpoint 602 is not configured with similar text communicationcapabilities as the video endpoint 602.

The functions discussed above with reference to the video endpoints 602may be carried out, in large part, by the control circuit 644 of eachvideo endpoint 602. In addition, the control circuit 644 may beconfigured to provide a user interface (e.g., the graphical userinterface 800 illustrated in and discussed with reference to FIGS. 8Athrough 8D) to enable the audibly-impaired user to use the videoendpoint 602. In some embodiments, the control circuit 644 may include amicrocontroller, a central processing unit, a field programmable gatearray, an application specific integrated circuit (ASIC), a programmablelogic controller (PLC), a system on chip, other circuits, orcombinations thereof.

In some embodiments, the control circuit 644 may include a computingdevice including at least one processing element 220 and at least onememory device 230, as illustrated in FIG. 2. The memory device 230 mayinclude computer-readable instructions directed at least to providingthe user interface. The processing element 220 may be configured toexecute the computer-readable instructions of the memory device 230.Accordingly, the computer-readable instructions transform the computingdevice from a computer into a special purpose computer configured forcarrying out embodiments of the disclosure. By way of non-limitingexample, the control circuit 644 may include a microcontrollerconfigured to execute computer-readable instructions stored internallyor externally to the microcontroller. Also by way of non-limitingexample, the control circuit 644 may include a central processing unit(CPU) (e.g., of a desktop personal computer (PC), a laptop computer, atablet computer, a smartphone, a PDA, etc.) configured to execute thecomputer-readable instructions in an operating system environment.

In some embodiments, the control circuit 644 may include an array oflogic circuits arranged and interfaced in a manner selected to performat least a portion of the functions the control circuit 644 isconfigured to perform. By way of non-limiting example, the controlcircuit 644 may include an array of logic circuits interfaced withprogrammable switches that are positioned in a manner selected to carryout the functions that the control circuit 644 is configured to perform(e.g., an FPGA). Also by way of non-limiting example, the array of logiccircuits may be interfaced using hard-wired interfaces (e.g., an ASIC).

FIG. 7 is a block diagram illustrating entries in the database 642 ofFIG. 6. The database 642 may be thought of as a list of text strings 710that may be selected and transmitted to other video endpoints 602 (FIG.6). While not illustrated, other fields may be included for eachdatabase entry, such as fields related to specific contacts, groups ofcontacts, various categories, and other information that may be usefulin filtering or selecting specific database entries.

Similar to the communication devices 102 (FIGS. 1 and 2) discussedabove, the video endpoints 602 may each include an input device 226(FIG. 2). The input device 226 may include a keypad, a keyboard, atouchscreen, a remote control, a mouse, other input device, orcombinations thereof configured to enable the user to initiate videocommunications with other users of other video endpoints 602 (e.g., byinteracting with an electronic contacts list, entering uniqueidentifiers corresponding to other video endpoints 602, etc.). Inaddition, the user may be able to interact with the database 642 usingthe input device 226. For example, the user may create and add new textstrings 710 to the database 642, or delete or modify existing textstrings 710 in the database 642.

As non-limiting examples, the database 642 illustrated in FIG. 7includes text strings 710 including the user's telephone number,address, favorite pizza order, utility account number, email address,and driving instructions to the user's place of residence.

In some embodiments, the text strings 710 may be categorized andsortable by category. By way of non-limiting example, a category may becreated for the user's account numbers, and all the text strings 710including account numbers may be categorized in the account numberscategory. Also by way of non-limiting example, a category may be createdfor the user's contact information, and all the text strings 710including the user's contact information (e.g., phone number,residential address, work address, email address, etc.) may becategorized in the contact information category. In some embodiments, asingle text string 710 may be categorized in more than one category.

FIGS. 8A through 8D are examples of a simplified graphical userinterface 800 of a video endpoint 602 of FIG. 6. The graphical userinterface 800 may be configured to enable a user to communicate usingtext, and access the database 642 (FIGS. 6 and 7) during a video callwith another user of another video endpoint 602. The graphical userinterface 800 may include a video field 810 for displaying video (e.g.,of the other user) from video data 608 received from the other videoendpoint 602. The graphical user interface 800 may also include a textediting field 830 configured to enable a user to input and modify text.Text in the text editing field 830 may be displayed by the other videoendpoint 602 to the other user of the other video endpoint 602 duringvideo communications with the other video endpoint 602. In other words,the video endpoint 602 may send text data 609 (or text embedded into thevideo 608) including the text in the text editing field 830 to the othervideo endpoint 602 for display by the other video endpoint 602.

The graphical user interface 800 may also be configured to enable theuser to select from text strings 710 saved in the database 642. Forexample, responsive to the user selecting a dropdown selection 820, thegraphical user interface 800 may present a list 860 of the text strings710 saved in the database 642, as shown in FIG. 8B. The order in whichthe text strings 710 are displayed may be based upon the number of timeseach of the text strings 710 have been used, the order in which the textstrings 710 were added to the database 642, alphabetical order, a userdefined order, or some other ordering protocol. In some embodiments, theuser may be enabled to switch between the different ordering protocols.

Each of the text strings 710 displayed in the list 860 of the textstrings 710 may be user-selectable. Responsive to a user selecting oneof the text strings 710 from the list 860 of text strings 710, theselected text string 710 may be displayed in the text editing field 830,and transmitted by the video endpoint 602 to the other video endpoint602 for display by the other video endpoint 602.

In some embodiments, the list 860 of text strings 710 may be organizedin categories 850, as illustrated in FIG. 8C. The categories 850 may beuser-selectable. Responsive to a user selection of one of the categories850 the graphical user interface 800 may display a list 860 of the textstrings 710 that are categorized in the selected category 850. Forexample, FIG. 8C illustrates an “ALL” category as selected.Consequently, all the text strings 710 are displayed in the list 860 ofFIG. 8C.

In some embodiments, when the user begins typing text into the textediting field 830, the user interface may automatically list textstrings 710 saved in the database 642 that may match what the user istyping. The text strings 710 listed below the text editing field 830 maybe user-selectable, and a selected text string 710 may be displayed inthe text editing field 830. For example, as shown in FIG. 8D, theletters “Lar” are shown typed into the text editing field 830. As aresult, the user-selectable text string 710 “Large deep-dish withpepperoni, olives, and extra cheese” is listed below the text editingfield 830.

In some embodiments, one or more of the text strings 710 may beassociated with hotkeys (e.g., selections of a key or a combination ofmultiple keys using a keypad or a keyboard). In other words, the userinterface 800 may display the one of the text strings 710 in the textediting field 830 that is associated with a hotkey that has beenselected by the user. The graphical user interface 800 may also beconfigured to enable the user to assign hotkeys to the various textstrings 710.

The graphical user interface 800 may also be configured to enable theuser to save text from the text editing field 830 to the database 642 asa text string 710. For example, the graphical user interface 800 mayinclude a save selection 840. Responsive to a user selection of the saveselection 840, text from the text editing field 830 may be saved as atext string 710 to the database 642 (FIGS. 6 and 7). In someembodiments, the user may be prompted to select one or more categories850 to which the saved text string 710 should belong. Consequently, ifthe user finishes typing a text string 710 in the text editing field 830and decides that the text in the text string 710 may be useful in alater conversation, the user may save the text string 710 for later use.

Although not illustrated, the graphical user interface 800 may beconfigured to enable other functions. For example, the graphical userinterface 800 may be configured to enable the user to save over anexisting text string 710 (e.g., change the text string 710 and save thechanges). Also, the graphical user interface 800 may be configured toenable the user to delete text strings 710 from the database 642. Thegraphical user interface 800 may further be configured to enable theuser to create, remove, and modify text strings 710 while notparticipating in a video call with another user. In addition, thegraphical user interface 800 may be configured to enable the user tosave text received from the other video endpoint 602 to the database 642as text strings 710.

Although also not illustrated, the graphical user interface 800 mayinclude options that enable the audibly-impaired user to participate invideo communications. By way of non-limiting example, the graphical userinterface 800 may include options that enable the audibly-impaired userto place and answer video calls. Those of ordinary skill in the art willappreciate that the graphical user interface 800 may provide many otheroptions known in the art of video communications for audibly-impairedpersons.

FIG. 9 is a simplified flowchart 900 illustrating a method of operatinga video endpoint 602 of FIG. 6. Referring to FIGS. 6 and 9 together, atoperation 910, the method may include engaging in video communicationswith another video endpoint 602. Engaging in the video communicationswith the other video endpoint 602 may include transmitting and receivingvideo 608 to and from the other video endpoint 602.

At operation 920, the method may include enabling an audibly-impaireduser of the video endpoint 602 to select from text strings 710 (FIG. 7)stored in a database 642. In some embodiments, enabling theaudibly-impaired user to select from the text strings 710 may includeenabling the audibly-impaired user to select from text strings 710stored in a storage 640 of the video endpoint 602. In some embodiments,enabling the audibly-impaired user to select from the text stringsincludes accessing a storage 662 of a server 660. In some embodiments,enabling the audibly-impaired user of the video endpoint 602 to selectfrom the text strings 710 includes providing a user interface, such as,for example, the graphical user interface 800 of FIGS. 8A through 8D.

At operation 930, the method may include transmitting data correspondingto a selected one of the text strings 710 stored in the database 642 tothe other video endpoint 602 during the video communications for displayby the other video endpoint 602 responsive to the audibly-impaired userselecting the selected one of the text strings 710. In some embodiments,transmitting the data corresponding to the selected one of the textstrings 710 may include transmitting separate text 609 and video 608 tothe other video endpoint 602. In some embodiments, transmitting the datacorresponding to the selected one of the text strings 710 may includetransmitting video 608 with the text embedded therein. In someembodiments, transmitting the data corresponding to the selected one ofthe text strings 710 may include transmitting the data using a real-timetext protocol. In some embodiments, transmitting the data correspondingto the selected one of the text strings 710 may include transmittingtext 609 to the other video endpoint 602 if it is determined that theother video endpoint 602 is capable of receiving text 609, andtransmitting video 608 with the text embedded therein if it isdetermined that the other video endpoint 602 is not capable of receivingtext 609.

Returning to operation 910, during the video communications with theother video endpoint 602, at operation 940 the method may includereceiving data corresponding to text from the other video endpoint 602.In some embodiments, receiving the data corresponding to the text fromthe other video endpoint 602 may include receiving separate text 609 andvideo 608 from the other video endpoint 602. In some embodiments,receiving the data corresponding to the text from the other videoendpoint 602 may include receiving text embedded into the video 608.

At operation 950, the method may include displaying the textcorresponding to the data received from the other video endpoint 602. Insome embodiments, displaying the text may include displaying the text onan electronic display 224 (FIG. 2) of the video endpoint 602. In someembodiments, displaying the text may include displaying the text in atext editing field 830 of a graphical user interface 800 (FIG. 8). Insome embodiments, displaying the text may include displaying the text ina video field 810 of the graphical user interface 800.

FIG. 10 is a simplified block diagram of a communication system 1000 fortransforming a computing device 1010 into a video endpoint 602 of FIG.6. The communication system 1000 may include the computing device 1010and one or more servers 1020 (hereinafter referred to as “servers”1020). The servers 1020 may include one or more data storage devices1022 (hereinafter referred to as “storage” 1022). The storage 1022 mayinclude computer-readable instructions directed to instructing thecomputing device 1010 to perform the functions discussed herein withreference to the video endpoint 602 of FIG. 6. For example, thecomputer-readable instructions may be directed to instructing thecomputing device 1010 to provide a user interface to an audibly-impaireduser (e.g., the graphical user interface 800 of FIGS. 8A through 8D) toenable the audibly-impaired user to select from text strings 710 storedin a database 642 (FIG. 7) and transmit a selected one of text strings710 to another video endpoint 602 for presentation to another user.

The servers 1020 may be configured to communicate with the computingdevice 1010 (e.g., through the Internet). By way of non-limitingexample, the servers 1020 may be configured to provide an online mobilesoftware application store, and the computing device 1010 may include amobile device configured to access the mobile software applicationstore. The servers 1020 may transmit data corresponding to thecomputer-readable instructions to the computing device 1010. Thecomputing device 1010 may be configured to execute the computer-readableinstructions. Accordingly, the computing device 1010 may be transformedinto a video endpoint 602.

Also by way of non-limiting example, the servers 1020 may be configuredto provide computer-readable instructions that may be executed by adesktop or laptop computer in an operating system environment. Forexample, the computer-readable instructions may be configured assoftware applications that may be executed in the operating systemenvironment. As a further example, the computer-readable instructionsmay include one or more web applications that may be executed throughthe computer's web browsing software (e.g., using javascripts). Otherconfigurations for the delivery of the computer-readable instructions tothe computing device 1010 are also contemplated within the scope of thedisclosure.

FIG. 11 is a simplified flowchart 1100 illustrating a method oftransforming the computing device 1010 of FIG. 10 into a video endpoint602 of FIG. 6. Referring to FIGS. 6, 10, and 11 together, at operation1110, the method may include storing computer-readable instructions on astorage 1022 of the servers 1020. The computer-readable instructions maybe directed to instructing the computing device 1010 to provide a userinterface to an audibly-impaired user. The user interface may beconfigured to enable the audibly-impaired user to communicate withanother user of another video endpoint 602 with text during videocommunications with the other user, and enable the audibly-impaired userto select from text strings 710 (FIG. 7) stored in a database 642 (FIG.6). The computer-readable instructions may also be directed toinstructing the computing device 1010 to transmit a selected one of thetext strings 710 to the other video endpoint 602 for presentation to theother user.

At operation 1120, the method may include transmitting thecomputer-readable instructions to the computing device 1010.Transmitting the computer-readable instructions may include transmittinga mobile software application, a software application configured forexecution in a computer operating system environment, web applicationscripts, other computer-readable instruction configurations, andcombinations thereof.

While certain illustrative embodiments have been described in connectionwith the figures, those of ordinary skill in the art will recognize andappreciate that embodiments encompassed by the disclosure are notlimited to those embodiments explicitly shown and described herein.Rather, many additions, deletions, and modifications to the embodimentsdescribed herein may be made without departing from the scope ofembodiments encompassed by the disclosure, such as those hereinafterclaimed, including legal equivalents. In addition, features from onedisclosed embodiment may be combined with features of another disclosedembodiment while still being encompassed within the scope of embodimentsencompassed by the disclosure as contemplated by the inventors.

What is claimed is:
 1. A video communication system configured tofacilitate communication between a first user and a second user throughassistance of a call assistant, the communication system comprising: afirst video communication device associated with the first user; and avideo relay service associated with the call assistant and configured tofacilitate a communication session between the first video communicationdevice and a second communication device associated with the seconduser, the communication session including: a first communication linkfor transmitting at least video data between the first videocommunication device and the video relay service; and a secondcommunication link for transmitting at least audio data between thevideo relay service and the second communication device, wherein thefirst video communication device is configured to: generate a graphicaluser interface displayed by an electronic display of the first videocommunication device during the communication session; retrieve textstrings during the communication session from a database of pre-storedtext strings created by the user that are categorized and sortablewithin the database by different subject matter categories andselectable by the user, wherein the first video communication device isconfigured to retrieve the text strings responsive to the user selectingone of the different subject matter categories through the graphicaluser interface; display the retrieved text strings on the graphical userinterface during the communication session; receive an input from theuser through the graphical user interface selecting one of the displayedtext strings during the communication session; modify the selected textstring responsive to a user input through the graphical user interfaceduring the communication session; transmit the modified selected textstring to the video relay service during the communication session fordisplay by an electronic display of the video relay service; and storethe modified selected text string in the database of pre-stored textstrings available for future selection during a subsequent communicationsession.
 2. The video communication system of claim 1, wherein one ormore text strings may be categorized and sortable within the database bymore than one of the different subject matter categories.
 3. The videocommunication system of claim 1, further comprising a remote serverassociated with the video relay service and configured to store thedatabase.
 4. The video communication system of claim 3, wherein thefirst video communication device is configured to retrieve at least aportion of the database from the remote server.
 5. The videocommunication system of claim 4, wherein the first video communicationdevice is configured to store the portion of the database locally withina data storage device of the first communication device.
 6. The videocommunication system of claim 4, wherein the first video communicationdevice is configured for the user to further modify text strings storedin the database stored within the remote server when not engaged in anycommunication session.
 7. The video communication system of claim 6,further comprising additional video communication devices associatedwith the first user, wherein the remote server is further configured toshare the modified text strings received from the first videocommunication device with the additional video communication devicesassociated with the first user.
 8. The video communication system ofclaim 1, wherein the first video communication device is configured tomodify the database of text strings through the graphical user interfaceby adding new text strings, deleting the text strings, or updating thetext strings in the database for subsequent selection during thecommunication session or another communication session.
 9. The videocommunication system of claim 3, wherein the first user is anaudibly-impaired user, the second user is an audibly-capable user, andthe call assistant is a sign language interpreter.
 10. A method offacilitating a communication session for a video endpoint, the methodcomprising: establishing a first video communication session includingcommunicating video data between a first video endpoint and a secondvideo endpoint, wherein establishing the first video communicationsession includes: establishing a first communication link between thefirst video endpoint associated with an audibly-impaired user and thesecond video endpoint associated with call assistant of a video relayservice; and establishing a second communication link between the secondvideo endpoint and a third communication endpoint associated with anaudibly-capable user; displaying a graphical user interface on the firstvideo endpoint during the first video communication session; accessing adatabase of text strings during the first video communication sessionresponsive to a user input through the graphical user interface of thefirst video endpoint, the text strings pre-stored, categorized, andsortable within the database by different subject matter categoriesprior to the first video communication session; displaying one or moretext strings from the database on the graphical user interface of thefirst video endpoint during the first video communication sessionresponsive to a user selecting one of the different subject mattercategories through the graphical user interface of the first videoendpoint; selecting a text string from the displayed text stringsresponsive to a user input through the graphical user interface duringthe first video communication session without requiring the user toinput individual characters of the selected text string; modifying theselected text string responsive to a user input through the graphicaluser interface during the first video communication session;transmitting the modified selected text string from the first videoendpoint to the second video endpoint for display by the second videoendpoint during the first video communication session; and storing themodified selected text string in the database of text strings availablefor future selection during a subsequent video communication session.11. The method of claim 10, wherein transmitting the selected textstring includes transmitting text data using a real-time text protocol.12. The method of claim 10, wherein the first communication linkincludes a video connection, and the second communication link includesan audio connection.
 13. The method of claim 10, further comprising:displaying the modified selected text string on an electronic display ofthe second video endpoint associated with the call assistant of thevideo relay service; and transmitting an audio translation of themodified selected text string from the second video endpoint to thethird video associated with the audibly-capable user.
 14. The method ofclaim 10, further comprising modifying the database by at least one ofadding new text strings to the database, removing text strings from thedatabase, or modifying the text strings stored in the database.
 15. Themethod of claim 14, wherein the database is a shared database such thatchanges to the text strings in the database are available to other videocommunication devices that are operated by the same user.
 16. The methodof claim 10, wherein accessing the database of text strings during thefirst video communication session includes the first video communicationdevice locally accessing the text strings during the first communicationsession.
 17. The method of claim 10, wherein accessing the database oftext strings during the first video communication session includes thefirst video communication device remotely accessing the text stringsduring the first communication session from a remote server comprisingthe database.
 18. The method of claim 10, wherein transmitting themodified selected text string from the first video endpoint to thesecond video endpoint includes embedding the text selected text stringinto video data transmitted from the first video endpoint to the secondvideo endpoint.
 19. The method of claim 10, further comprising:receiving the modified selected text string at the second video endpointfor display by the second video endpoint during the first videocommunication session; modifying the displayed text string at the secondvideo endpoint; and storing the modified text string in a database oftext strings associated with the second video endpoint.
 20. The videocommunication system of claim 1, wherein the first video communicationdevice is further configured to enable the user to assign a hotkey to atleast one text string of the database of pre-stored text strings suchthat the assigned text string is transmitted to the video relay serviceduring the communication session responsive to the hotkey beingselected.